The TextClass Benchmark project is an ongoing, continuous benchmarking process that aims to provide a comprehensive, fair, and dynamic evaluation of LLMs and transformers for text classification tasks. This evaluation spans various domains and languages in social sciences disciplines engaged in NLP and text-as-data approach. The leaderboards present performance metrics and relative ranking using a tailored Elo rating system. With each leaderboard cycle, novel models are added, fixed test sets can be replaced for unseen, equivalent data to test generalisation power, ratings are updated, and a Meta-Elo leaderboard combines and weights domain-specific leaderboards. This article presents the rationale and motivation behind the project, explains the Elo rating system in detail, and estimates Meta-Elo across different classification tasks in social science disciplines. We also present a snapshot of the first cycle of classification tasks on incivility data in Chinese, English, German and Russian. This ongoing benchmarking process includes not only additional languages such as Arabic, Hindi, and Spanish but also a classification of policy agenda topics, misinformation, among others.

Large language models are typically trained on a wide and diverse distribution of text. For example, a "base" or "foundation" model may simultaneously be trained to write news articles, 1990s-era DHTML, and impassioned forum discourse on fictional character romances. While such wide-ranging modeling capabilities are fascinating, they often prove difficult to control for the average user. The release of ChatGPT (and its myriad later offspring) has popularized the "chat" paradigm for interacting with large language models, which imbues a base model with steerability by training it to adopt the persona of a helpful assistant - the "chatbot". A more general version of the chat-tuned model is the instruct-tuned model [24, 33], where the base model is trained to respond to imperative statements, e.g. What are some interesting places to visit in San Francisco?